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ABSTRACT 



Since 1992, the Vermont Department of Education has granted 
funds to the University of Vermont (UVM) to operate the Migrant Education 
Program (MEP) . This review by the State Auditor found that prior to fiscal 
year 1999, the Department did not make a concerted effort to oversee the MEP 
and, as a result, several critical programs components were not put into 
place as required by federal law. The Department did not provide adequate 
oversight of UVM's administrative costs. Together, UVM and the Department 
spent approximately 33 percent of the State's 1999 MEP grant award on 
administrative costs. The Department failed to compel UVM to meet MEP 
requirements, including development of a complaint process, creation of 
regional advisory boards, and monitoring of recruitment efforts. As a result, 
the program took an average of 158 days to enroll migrant children after a 
qualifying move. The MEP did not consult with parent advisory councils to 
solicit their input into the planning and operation of the program at both 
the state and local operating level . The MEP was not integrated with other 
mainstream educational programs and the state plan did not reflect 
integration with other educational programs as required by law. Program goals 
and outcomes for the MEP were not measurable, compromising the ability to 
evaluate the effectiveness of the program. Three appendices present analysis 
of program evaluations and performance measurements, analysis of program 
objectives and indicators, and the Department of Education's response to the 
draft review. (TD) 
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STATE AUDITOR’S REVIEW 

of the 

VERMONT MIGRANT EDUCATION PROGRAM 

July 1, 1997 through December 31, 1999 

Executive Summary 

The Office of the State Auditor has conducted a review of the Department of Education’s (the 
Department) Migrant Education Program (MEP), initiated as a result of complaints fi-om a vari- 
ety of sources related to changes to the service delivery model and administrative costs associ- 
ated with operating the program. 

Findings and Recommendations 

Our review found that the Department of Education’s Migrant Education Program and the 
state plan as submitted by the University of Vermont did not comply with federal law. 

■ The Migrant Education Program did not consult with parent advisory 
councils to solicit their input into the planning and operation of the pro- 
gram at both the State and local operating level; 

■ Program goals and outcomes for the Migrant Education Program were 
not measurable, compromising the Department’s and UVM’s ability to 
evaluate the effectiveness of the program. 

■ Changes in Migrant Education Program strategies and programs were 
not reflected in the Department’s state plan. 

■ The Migrant Education Program was not integrated with other main- 
stream educational programs and the state plan did not reflect integration with 
other educational programs as required by law. 

Our recommendation is that the Department of education comply with all laws and regulations in 
the development, maintenance and implementation of the MEP state plan. 

We also found that the Department did not have an internal control process adequate to 
provide reasonable assurance that it fulfilled its monitoring and oversight responsibilities 
or that it complied with applicable federal laws. 

■ The Department did not provide adequate oversight of UVM’s administrative costs. 
Together, UVM and the Department spent approximately 33% of the 
State’s FY 1999 Migrant Education Program grant award on adminis- 
trative costs; 
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■ The Department failed to compel UVM to meet MEP program requirements to develop a 
complaint process, to develop regional advisory boards and to monitor recruitment ef- 
forts; as a result it took an average of 158 days to enroll migrant chil- 
dren in the program after a qualifying move; and, 

■ The Department did not take an active role in the development of the 
Migrant Education Program state plan which was not in compliance 
with federal law. 

Our recommendation is that the Department should develop and implement a system of internal 
controls to provide reasonable assurance that adequate monitoring and oversight of contracted 
MEP activities will be provided and that the Department complies with applicable federal laws. 

Background 

Congress created the Migrant Education Program as part of the Elementary and Secondary Edu- 
cation Act of 1965 (ESEA). The MEP was instituted in response to concerns about the welfare 
and educational disadvantages of the children of migrant farm workers. 

According to federal law, the purposes of the MEP are to help migrant children by ensuring that 
the services provided: 

• meet the same challenging state content standards and performance standards that all chil- 
dren are expected to meet and that the children benefit from state and local systemic re- 
forms; 

• help reduce the educational disruptions and other problems that result from repeated 
moves; 

• address the special needs of migrant children in a coordinated and efficient manner; and, 

• help the children overcome educational disruption, cultural and language barriers, social 
isolation, various health-related problems, and other factors that inhibit their ability to do 
well in school, and to prepare them to make a successful transition to post-secondary edu- 
cation or employment. 

Federal law and regulation define an eligible child as one who is (or whose parent, spouse, or 
guardian is) a migratory agricultural, dairy or fisheries worker who, in the preceding 36 months, 
has moved from one school district to another (or accompanied a parent, spouse or guardian) in 
order to obtain temporary or seasonal employment. A child is eligible for MEP services if he or 
she is between the ages of three and 22 and has not graduated from high school and does not 
hold a high school equivalency certificate. Children who are too young to attend school- 
sponsored educational programs but who are old enough to benefit from an organized instruc- 
tional program also are eligible. 

Before services can be provided, MEP staff must complete a certificate of eligibility for a family. 
Eligibility is determined by a combination of factors including whether the work of the parent(s) 
qualifies, whether the move to the new school district is temporary, and whether arrival and resi- 
dency dates meet program requirements. 
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Profile of Students and Families 



As of May 1999, there were approximately 1,000 children in Vermont who were receiving MEP 
services. About 65 percent of these students resided in Orleans, Franklin, Caledonia, and 
Addison counties and approximately one-quarter of the children were of pre-school age. In 
1997, the MEP estimated that 39 percent of parents of children receiving services at that time 
had not completed high school and that 62 percent of the children were associated with poverty 
risk factors. 

Funding and Operations 

Funding for the program began in 1969. Over time, however. Congress expanded the criteria for 
eligibility to include work in dairy, logging, fisheries and certain types of food processing, which 
allowed New England and Midwestern states to access more federal funding. In 1994, Ver- 
mont’s aimual grant peaked at more than $1 million. 

The funds are granted to the state departments of education, which may operate the programs or 
contract with other agencies or organizations. In the early years, Vermont’s MEP was operated 
by the Department of Education (the Department) out of its Montpelier offices. As the program 
grew, the MEP moved its administrative offices to the South Burlington Middle School. Pro- 
gram staff consisted of administrators and numerous part-time teachers who provided a variety 
of services to migrant families and their children. In 1992, the Department contracted with the 
University of Vermont (UVM) to operate the program and MEP became part of the College of 
Education and Social Services’ Office of Rural Education. Initially, there were few changes to 
the program. It remained headquartered in South Burlington and the Program’s Director was 
retained by UVM. The MEP relocated its offices to the UVM campus in 1997. 

When Congress re-authorized the ESEA in 1994, the term of student eligibility was reduced 
fi"om six years duration to three. As a result, the caseload declined somewhat and, because 
funding is based largely on caseload, Vermont’s allocation dropped fi"om a high of $1.14 million 
in 1994 to $765,000 in 1999, a decline of 33 percent in five years. As a response, the MEP 
changed its strategy in the fall of 1996 by reducing field staff and modified its programs to em- 
phasize home-based assistance rather than primarily in-school programs. 

Services 

Depending on an assessment of the needs of the family, services offered during the review period 
included direct instruction (in school or at home), coordination of various social and health 
services, preschool and summer programs, assistance with the transfer of records, guidance and 
counseling for parents and high school dropouts (e.g., referrals to General Equivalency Diploma 
programs), and advocacy on behalf of students for other special education services for which 
they may have qualified. 

Complaints and Program Review 

The program change fi"om school-based programs to home-based assistance led to complaints by 
some parents, former employees, and school officials, and gained attention fi-om the news media, 
State legislators, members of Vermont’s congressional delegation, and the Department. The 
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complaints focused on the changes to the service delivery model and UVM’s administrative 
costs. 

As a result of its internal review, the Department has chosen to contract with a group of supervi- 
sory unions in the Northeast Kingdom to operate the MEP in that area. According to Department 
staff a second consortium of supervisory unions organized to provide MEP services in Franklin 
County. By the end of FY 2000, the Department plans to transition all services from UVM to 
local consortia or individual supervisory unions. 
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STATE AUDITOR’S REVIEW 

of the 

VERMONT MIGRANT EDUCATION PROGRAM 

July 1, 1997 through December 31, 1999 

I. PURPOSE, SCOPE AND METHODOLOGY 



The Office of the State Auditor has conducted an internal control and compliance review of the 
Vermont Migrant Education Program (“MEP”). The purpose of the review was to assist the 
Vermont Department of Education’s management (the Department) in achieving the goals of the 
Elementary and Secondary Education Act of 1965, Part C, Migrant Education (re-authorized in 
1994 by the Improving America’s Schools Act, P.L. 103-382). 

This review was conducted in accordance with this Office’s responsibilities and authority con- 
tained in 32 V.S.A. §§ 163 and 167 and as part of the State Auditor’s aimual 0MB Circular A- 
133 A Single Audit. The scope of the review included an evaluation of the operation, perform- 
ance, and financial management of the MEP, including internal control policies and compliance 
with relevant statutes, regulations, and contracts. 

A review differs substantially from an audit conducted in accordance with applicable profes- 
sional standards. The purpose of an audit is to express an opinion. The purpose of a review is to 
identify findings and make recommendations so that the reviewed agency can better accomplish 
its mission and more fully comply with laws and regulations. This review relied upon represen- 
tations of, and information provided by the Department and MEP staff. If an audit had been per- 
formed, the findings and recommendations may or may not have differed. 

The methodology included a review of relevant statutes, regulations, grant applications, con- 
tracts, financial data, evaluations, internal memoranda, and correspondence, as well as numerous 
phone calls with Department and MEP staff. 



II. BACKGROUND 



Congress created the Migrant Education Program as part of the Elementary and Secondary Edu- 
cation Act of 1965 (ESEA). The MEP was instituted in response to concerns about the welfare 
and educational disadvantages of the children of migrant farm workers. As the U.S. General Ac- 
counting Office noted in a 1998 report, “poverty, limited English ability, and rural and social 
isolation place children in migrant and seasonal work - like any other group of children affected 
by these social conditions - at considerable risk of academic failure. For these children, how- 
ever, the difficulties associated with these social conditions are compounded by mobility and 
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other conditions of agricultural work that result in school enrollment rates and high school com- 
pletion rates among the lowest in the nation.”* 

A. Program Purposes 

According to federal law, the purposes of the MEP are tor 

• “Ensure that migrant children have the opportunity to meet the same challenging state 
content standards . . . and performance standards that all children are expected to meet; 

• Support high quality and comprehensive educational programs for migratory children to 
help reduce the educational disruptions and other problems that result from repeated 
moves; 

• Ensure that migrant children are provided with appropriate educational services (including 
supportive services) that address their special needs in a coordinated and efficient manner; 

• Design programs to help migrant children overcome educational disruption, cultural and 
language barriers, social isolation, various health-related problems, and other factors that 
inhibit the ability of such children to do well in school, and to prepare such children to 
make a successful transition to post-secondary education or employment; and 

• Ensure that migrant children benefit from state and local systemic reforms.”^ 

B. Eligibility Definitions 

Federal law and regulation define an eligible child as one who is (or whose parent, spouse, or 
guardian is) a migratory agricultural, dairy or fisheries worker who, in the preceding 36 months, 
has moved from one school district to another (or accompanied a parent, spouse or guardian) in 
order to obtain temporary or seasonal employment. Such a child is eligible for MEP services if 
he or she is between the ages of three and 22 and has not graduated from high school and does 
not hold a high school equivalency certificate. Children who are too young to attend school- 
sponsored educational programs but who are old enough to benefit from an organized instruc- 
tional program also are eligible. 

Before services can be provided, MEP staff must complete a certificate of eligibility for a family. 
Eligibility is determined by a combination of factors including whether the work of the parent(s) 
qualifies, whether the move to the new school district is temporary, and whether arrival and resi- 
dency dates meet program requirements. 

C. Profile of Students and Families 

As of May 1999, there were approximately 1,000 children in Vermont who were receiving MEP 
services. About 65 percent of these students resided in Orleans, Franklin, Caledonia, and 
Addison counties and approximately one-quarter of the children were of pre-school age.^ In 
1997, the MEP estimated that 39 percent of parents of children receiving services at that time 



' “Child Labor In Agriculture: Changes Needed to Better Protect Health and Educational Opportunities,” U.S. 
General Accounting Office, August 1998, p. 49. 

^ Preliminary Guidance for Migrant Education Program, Tide I, Part C, Public Law 103-382, U.S. Department of 
Education, June 1998, p. viii. 

^ Improving America’s School Act, P.L. 103-382, §1309(2), and 34 CFR §200.40(c) and (e). 

'* May 28, 1999 telephone conversation between Doug Hoffer, Consultant, KPMG contractor and Shirley Wolfe, 

Acting MEP Director. 
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had not completed high school and that 62 percent of the children were associated with poverty 
risk factors/ 

D. Funding and Operations 

Although funding for the program began in 1969, Vermont received very little money in the 
early years because few families qualified for assistance. Over time, however. Congress ex- 
panded the criteria for eligibility to include work in dairy, logging, fisheries and certain types of 
food processing, which allowed New England and Midwestern states to access more federal 
funding. In 1994, Vermont’s annual grant peaked at more than $1 million. 

The funds are granted to the state departments of education, which may operate the programs or 
contract with other agencies or organizations. In the early years, Vermont’s MEP was operated 
by the Department of Education (the Department) out of its Montpelier offices. As the program 
grew, the MEP moved its administrative offices to the South Bmlington Middle School. Pro- 
gram staff consisted of administrators and numerous part-time teachers who provided a variety 
of services to migrant families and their children. 

In 1992, the Department contracted with the University of Vermont (UVM) to operate the pro- 
gram and MEP became part of the College of Education and Social Services’ Office of Rmal 
Education. Initially, there were few changes to the program. It remained headquartered in South 
Bmlington and the Program’s Director (the only one the MEP had ever had) was retained by 
UVM. The MEP relocated its offices to the UVM campus in 1997. 

When Congress re-authorized the ESEA in 1994, the term of student eligibility was reduced 
fi-om six years dmation to three. As a result, the caseload declined somewhat and, because 
funding is based largely on caseload, Vermont’s allocation dropped fi'om a high of $1.14 million 
in 1994 to $765,000 in 1999, a decline of 33 percent in five years. As a response, the MEP 
changed its strategy in the fall of 1996 by reducing field staff and modified its programs to em- 
phasize home-based assistance rather than primarily in-school programs. 

E. Services 

Depending on an assessment of the needs of the family, services offered during the review period 
included direct instruction (in school or at home), coordination of various social and health 
services, preschool and summer programs, assistance with the transfer of records, guidance and 
counseling for parents and high school dropouts (e.g., referrals to General Equivalency Diploma 
programs), and advocacy on behalf of students for other special education services for which 
they may have qualified. 

F. Complaints and Program Review 

The program change fi-om school-based programs to home-based assistance led to complaints by 
some parents, former employees, and school officials, and gained attention fi-om the news media. 
State legislators, members of Vermont’s congressional delegation, and the Department. The 
complaints focused on the changes to the service delivery model and UVM’s administrative 
costs. The Department responded in the following ways: 




^ FY 99 MEP grant application / State plan, p. 6. 
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• A Request for Proposals was issued in early 1998 for a pilot program of regional admini- 
stration of the MEP in the Northeast Kingdom. Neither of the two proposals submitted 
was accepted; 

• UVM’s FY 99 MEP grant application was approved hy the Department with several con- 
ditions that required the MEP to address some issues raised earlier hy parents and others; 
and, 

• The Department assigned a staff person to review the entire program in November 1998. 

As a result of its internal review, the Department has chosen to contract with a group of supervi- 
sory imions in the Northeast Kingdom to operate the MEP in that area. According to Department 
staff a second consortium of supervisory unions organized to provide MEP services in Franklin 
Coimty. By the end of FY 2000, the Department plans to transition all services from UVM to 
local consortia or individual supeiyisory unions.^ 



III. COMPLIANCE FINDING AND RECOMMENDATION 



Comprehensive State Plan 
Finding 1: 



The Department of Education’s state plan as submitted by the University of Vermont for 
the Migrant Education Program did not comply with federal law in several respects: a) it 
was not reviewed and revised on a timely basis to reflect changes in strategies and pro- 
grams, b) it did not reflect the program’s integration with other educational programs, c) it 
did not reflect consultation with parent advisory councils, and d) it did not specify measur- 
able program goals and outcomes. 

Discussion: 

Relevant federal law requires: 

■ That the state plan be periodically reviewed and revised to reflect changes in the State’s mi- 
grant education strategies and programs.’ 

■ That the state plan be integrated with other educational programs. * 

■ That the State’s application for an MEP grant include assurances that there is appropriate 
consultation with parent advisory coimcils in the planning and operations of the MEP at both 
the State and local operating agency level.® 



* December 30, 1999 letter from Mary Mulloy, Consultant, Vermont Department of Education, to Michael Clasen, 
Office of the State Auditor. As of December 1999, the administrative stmcture for the MEP has changed in that 
the Associate Dean of the College of Education and Social Services and Mary Mulloy are providing supervision 
over the UVM staff which consists of one program specialist/teacher, four teachers and a .63 FTE secre- 
tary/bookkeeper. 

’ Public Law 103-382 § 1306(a)(2)(B). 

* Improving America’s Schools Act of 1994 (lASA), Public Law 103-382, §1306(a)(l). This statute reauthorized 
and amended the Elementary and Secondary Education Act of 1965 (ESEA). 

^ Public Law 103-382 § 1304(c)(3). 
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■ That the state plan specify measurable program goals and outcomes.’® 

The Department’s application and state plan failed to comply with these requirements as fol- 
lows.” 

a. ^ Review and Revise State Plan 

During our review period there were two notable changes to the MEP that required that the state 
plan be reviewed and revised. These changes included a shift in the service delivery model fi-om 
school-based to home based services and a move fi’om a single local operating agency to multi- 
ple local operating agencies. The state plan was not revised to reflect these changes. 

b. l MEP Integration 

Upon review of the State plan, we were unable to identify any language that outlined how the 
Department planned to integrate the MEP with other educational programs under the Improving 
America’s Schools Act. 

c. ) Consultation with Parent Advisory Councils 

A 1995 evaluation of the MEP found that “there is no evidence that parents are involved directly 
in the design of student programs.”’^ Following this report, MEP staff conducted some work- 
shops on farm safety and school readiness, but it was not until 1997 that a parent advisory coun- 
cil was formed, in Newport. The council met six times in the two-year period 1997 - 1998 and 
had an average of six parents in attendance. A parent coordinator and one or two other staff per- 
sons represented the MEP. Following the complaints made by parents, the Assistant Director 
attended a few meetings, but the Director attended only one when the former national MEP Di- 
rector visited Vermont. The meetings were discontinued after May 1998. We found no evidence 
that any employees of the Department attended these meetings. 

The Friends of Migrant Education, formed in 1996, met in Febmaiy 1998, when there were four 
MEP parents present. A representative of the U.S. Department of Education also was present at 
this meeting. After the Department directed the MEP to establish regional advisory boards, a 
Northeast Kingdom regional group (Tri-County Friends of Migrant Education) met in November 
of 1998 and included several parents. Thus, although several meetings involving parents were 
held during the review period, no parental advisory councils played a significant role in either the 
planning or operations of MEP at either the state or local operating level. 



Public Law 103-382 §1306(a)(l)(D). 

" In the Department of Education’s response to the Office of the State Auditor’s Draft Report of the Migrant Edu- 
cation Program, the Department noted that the Federal Office of Migrant Education accepted the Department’s 
plan. March 24, 2000 letter fi-om David Wolk, Commissioner, Department of education to Michael Clasen, 
Deputy State Auditor. 

Meyers, H.W., “1993-94 Evaluation Report for the Migrant Education Program of the Office of Rural Educa- 
tion,’’ College of Education and Social Services, UVM, March 19, 1995 (revised Febmary 12, 1996), p. 14. 

The MEP has organized and convened working groups on several occasions over the past five years to re- 
evaluate program design. In each instance, the committees established for these purposes consisted almost en- 
tirely of professionals, including educators, school administrators. Department and MEP staff, and UVM faculty 
and staff. The Review and Design (R & D) Committee (1994-95) included only two parents (out of more than 
500 families statewide) but the “parent representatives had some difficulty in attending all the meetings and 
therefore their involvement [was] limited.”*^ The fact that meetings were held during working hours at UVM 
State Auditor’s Review of the Migrant Education Program Page 9 of 14 



d.) Measurable Goals and Outcomes 



The stated goals in the 1999 MEP state plan include: equitable treatment, continuity in educa- 
tional experience, assurance of an opportunity to enter school ready to leam, provision of a wide 
range of educational activities, provision of specially targeted educational services, 
empowerment of parents, provision of training for educators, and support for children through 
outreach and advocacy. These goals represent generalities more appropriate to a mission state- 
ment; the only measurable goal among them is to “reduce the incidence of dropouts and improve 
the graduation rate among migrant students”. 

An evaluation was conducted in 1998 under a contract between UVM and an outside consult- 
ant.^^ Based on our review, the evaluation suffered from a lack of independent data and the con- 
sultant made no effort to verify the information provided by staff. The lack of reliable data 
makes the findings questionable and provide little reliable information for managers and policy 
makers. 

We found no evidence that the Department had any involvement in or reaction to this evaluation. 
Nor is there evidence that the Department has required the MEP to develop and implement a 
meaningful performance measurement plan as required by law.^^ 

Recommendation 1: 

The Department of Education must comply with federal law with regard to the Migrant 
Education Program application and state plan by revising the plan to reflect changes in 
migrant education strategies, describing how the program will be integrated with other 
educational programs, assuring consultation with parent advisory councils in the planning 
of migrant education activities, and specifying measurable program goals and outcomes. 



may have contributed to limited involvement by parents since most MEP families live in rural areas some dis- 
tance from Burlington. The recommendations in the 1995 report of the R & D Committee contained no mention 
of increased parental involvement in planning. Following the report of the R & D Committee in December 
1995, the MEP organized the Friends of Migrant Education, which met twice in early 1996 to continue the pro- 
gram redesign effort. This group was somewhat larger than the R & D Committee but included no parent repre- 
sentatives. 

An earlier evaluation was conducted in 1995 by UVM’s College of Education and Social Services. The method- 
ology included interviews with a small sample of students, their parents, teachers and school administrators, as 
well as a review of selected school records. The report noted that because “learning and behavioral objectives 
are of necessity varied and unstandardized ... norm referenced standardized tests are inappropriate measures of 
pupil outcomes. To the extent that they are used by state officials to index program success they will result in 
incomplete and flawed conclusions of program performance.” Additionally, “the fact that the program is de- 
signed to be a short-term, supplemental intervention mitigates to some extent against longitudinal design for 
program evaluation with more standard measures.” The report recommended against evaluating MEP students 
using the standardized tests that are now required throughout the State. The report suggested that it would be 
more appropriate “to continue independently conducted case studies of randomly drawn cases in order to verify 
the levels of performance.” The goal in the State plan at the time was that “80 % of the students will meet at 
least 75 % of the plaimed objectives set for them.” The researchers found that “the case studies and record re- 
views [were] sufficient to estimate in general that as many or more than two thirds of the students made impor- 
tant progress on educational plans.” However, there was a “lack of specific objectives, criteria, and measures in 
about a third of student records [so] it [was] not possible to evaluate the objective as written.” The 1995 study 
did not examine or evaluate the administrative structure, budget, or efficiency of the MEP. 

See Appendix A for a detailed analysis on Program Evaluation and Performance Measurement and Appendix B 

for an analysis of Program Objectives and Indicators. 
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IV. INTERNAL CONTROL FINDING AND RECOMMENDATION 



Finding 2; 

The Department did not have an internal control process adequate to provide reasonable 
assurance that it fulfilled its monitoring and oversight responsibilities or that it complied 
with applicable federal laws. 

Discussion: 

Internal control consists of five interrelated components including control environment, risk as- 
sessment, control activities, information and communication, and monitoring.*® The compliance 
finding above, as well as the findings outlined below, demonstrate that the Department of Edu- 
cation had an inadequate system of internal controls. 

A. Control Environment sets the tone of an organization, influencing the control consciousness 
of its people. Evidence of inadequate internal control processes for this component include 
the following: 

Lack of Department Involvement in Development of State Plan As the recipient of federal 
MEP fimds, the Department had a responsibility to perform oversight and monitoring of 
grant-fimded activities. Before November 1998, the Department appears to have exercised 
little control of the MEP. The law*’ requires that each state ensure that it and its local oper- 
ating agencies identify and address the education needs of migratory children in accordance 
with a state plan. The Department did not take an active role in the development of the state 
plan, relying instead upon UVM’s grant application to serve as the state MEP plan.*^ As 
noted in Finding 1 , this plan failed to comply with federal law. 

B. Risk Assessment is the entity’s identification, analysis, and management of relevant risks to 
achievements of its objectives. Relevant risks to the MEP include: 

Funding : Cutbacks in federal support for the MEP created significant problems for the pro- 
gram. These cutbacks led to a reduction in field staff and a multi-year effort to consider new 
or revised program design and delivery options. We found no evidence that program admin- 
istrators considered any significant changes in the administration of the program (e.g., ad- 
ministrative structiure and costs, affiliation with UVM vs. decentralization, etc.) until the De- 
partment began a strategic review of the program in 1997. 



American Institute of Certified Public Accountants, Statement on Auditing Standards (SAS) No. 78, Journal of 
Accountancy, February, 1996, pp. 85-90. 

’’ Improving America’s Schools Act of 1994 (lASA), Public Law 103-382, §1306(a)(l). This statute reauthorized 
and amended the Elementary and Secondary Education Act of 1965 (ESEA). 

According to 1997-1998 agreement between the Department and UVM, “the aimual state plan shall be developed 
by” UVM’s Office of Rural Education, submitted to the Department for approval and further submitted to the 
United States Department of Education. Other than a budget request, no State plan was submitted for FY 98. 
According to Department staff, the FY 97 UVM application was actually a two-year plan and the FY 98 UVM 
application became an amendment. However, there is no indication in the document that the FY 97 plan was 
intended to cover FY 97 and FY 98. 
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Administrative Costs : According to its FY 1999 grant, the Department required UVM to, 

“By September 18 [1998], re-evaluate MEP administrative structure and t^e steps to reduce 
administrative costs so as to be commensurate with migrant programs in states with similarly 
funded MEPs and educational programs in Vermont.”* 

The MEP has been operated by UVM for almost seven years. We foimd no evidence that the 
Department had ever questioned UVM’s administrative costs xmtil complaints were made 
about the allocation of resources. The MEP is funded entirely through federal grants. The 
MEP sought information about the structure and costs of MEPs in five other states but was 
unable to make useful comparisons. This was reported to the Department on November 16, 
1998, which was two months after the deadline imposed by the Department.^* 

The Department did not require UVM or the MEP to utilize any particular methodology for 
allocating grant funds to cover administrative costs. Although MEP staff prepared a one-page 
summary of administrative costs, the methodology was not based on federal Education De- 
partment General Administration Regulations.^^ 

UVM’s annual grant application to the Department includes a proposed budget that catego- 
rizes all expenditures as either direct or indirect, with the latter representing UVM’s 8 percent 
charge for administering an outside grant program. The so-called direct costs include all 
budget categories but the line items do not distinguish between administrative and program 
costs.^^ Based on a review of data provided by the UVM Controller’s Office and the MEP, it 
appears that administrative costs accoimted for up to one third of the total budget. Finally, 
UVM’s 8 percent charge for indirect costs is not based on the actual costs of administering the 
grant but is capped by the federal government. In FY 1999, the indirect charge was $68,674. 
Combined with the $51,103 expended by the Department, identifiable administrative costs 
totaled $1 19,777, or 16 percent of the FY 1999 federal grant award. Additional administra- 
tive costs imbedded in the direct costs would increase this percentage to approximately 33 
percent of the FY 1999 federal grant award. We requested an accoimting of the actual cost to 
UVM for the MEP, as well as a breakdown of the actual line item expenditures, but were not 
supplied with the information.^'* 

C. Control Activities are the policies and procedures that help ensure that management direc- 
tives are carried out. Examples where there was a breakdown in or absence of procedures to 
follow through on Department requirements include: 

Lack of Complaint Process According to its grant with UVM, the Department required that, 
“By October 15, 1998, [the MEP shall] establish and activate a simple multi-level complaint 



August 31, 1998 letter from Dave Baroudi, State Director of Title I Programs, Department of Education, to Jerry 
Robinson, MEP Director, University of Vermont. 

States have considerable flexibility in the implementation of migrant programs and in the services offered. 
November 16, 1998 letter from Shirley Wolfe, MEP Assistant Director, to Dave Baroudi, State Director, Title I 
Programs, Department of Education. 

Based on a telephone conversation between Mary Mulloy, Consultant, Vermont Department of Education and 
Doug Hoffer, Consultant, KPMG contractor, this view was shared by Department staff. 

“ See UVM-MEP FY 99 Grant Application, pp. 1 & 2. 

The request was made in a March 25, 1999 letter from Doug Hoffer, Consultant, KPMG contractor to Steve 
Stoddard, UVM Assistant Controller. The April 22, 1999 response included a brief description of indirect costs 

but no data specific to the MEP. 
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process / procedure for migrant families and MEP staff” As of April 20, 1999,^® no formal 

complaint process / procedure had been adopted by UVM, more than six months after the 
deadline imposed by the Department.^’ Additionally, we found no evidence that the Depart- 
ment took any steps to compel the UVM to comply with the requirement that the process be 
implemented on schedule. 

J.ack of Regional Advisory Boards According to its grant with UVM, the Department re- 
quired that, “By January 15, 1999, [the MEP shall] establish a regional advisory board (or 
boards) outside the Northeast region which [should] include equal representation fi-om fami- 
lies, schools, and social service agencies. This / these advisory board(s) will advise the Ver- 
mont MEP on policies and procedures, communications, etc.” As of April 20, 1999, no re- 
gional advisory boards had been established, more than three months after the deadline im- 
posed by the Department.’^ Additionally, we found no evidence that the Department took 
any steps to compel the MEP to comply with the requirement that the boards be established 
on schedule.’^ 

Untimely Recruitment Efforts According to its FY 1999, the Department required UVM to, 
“By May 31, 1999, and as part of an annual requirement, monitor and report to the Depart- 
ment on a recruitment indicator. One such indicator might be how much time elapses be- 
tween the move that establishes the eligibility and enrollment in the program.”’* UVM pro- 
vided the requested information within the established time fi’ame, however the data indicated 
that the average time elapsed between a qualitying move and enrollment in the program was 
158 days. In seven of twelve counties, the maximiun time elapsed was over 1 8 months. The 
Department’s oversight of UVM’ s recruitment efforts and the effectiveness of school proto- 
cols for identifying MEP-eligible new students was lacking. We found no evidence to support 
conclusions about whether recruitment has improved or declined. 

D. Information and Communication are the identification, capture and exchange of informa- 
tion in a form and time fi’ame that enable people to carry out their responsibilities. 

Evidence of a breakdown in the Department’s information and communication processes in- 
clude the Department’s failure to follow-up on specific time sensitive contractual require- 
ments that were not met by UVM. 
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August 31, 1998 letter from Dave Baroudi, State Director of Title I Programs, Department of Education, to Jerry 
Robinson, MEP Director, University of Vermont. 

April 20, 1999 letter from Mary Mulloy, Consultant, VT DOE to Doug Hoffer, Consultant, KPMG contractor. 

In this letter, Ms. Mulloy referred to an attachment from the MEP that described their continuing efforts as “the 
current status of the complaint process,” 

The only relevant correspondence provided by the VT DOE was a January 12, 1999 letter from the MEP Direc- 
tor that referred to a VT DOE review of a draft on September 1, 1998. The letter included an attached “working 
draft” for review and comment. 

August 31, 1998 letter from Dave Baroudi, State Director of Title I Programs, Department of Education, to Jerry 
Robinson, MEP Director, University of Vermont. 

April 20, 1999 letter from Mary Mulloy, Consultant, Vermont Department of Education to Doug Hoffer, Con- 
sultant, KPMG contractor. 

According to a April 20, 1999 letter from Mary Mulloy, to Doug Hoffer, as part of the Department’s strategic 
review of the program (and in preparation for the post-UVM MEP), Department staff have “been working with 
[UVM] to meet with superintendents in other areas of the state. 

August 31, 1998 letter from Dave Baroudi State Director of Title I Programs, Department of Education, to Jerry 

Robinson, MEP Director, University of Vermont. 
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The Department also failed to review and revise the state plan in light of significant adminis- 
trative and program changes, and failed to articulate measurable program goals and outcomes. 

E. Monitoring is a process that assesses the quality of internal control performance over time. 



Evidence of the Department’s failure to implement adequate monitoring processes include, as 
mentioned above. Department approval of state applications for federal fimds and state plans 
that did not meet a number of federal requirements, infi’equent UVM sponsored evaluations of 
the MEP that were deficient, high administrative costs relative to total fimds, budgets that did 
not separate administrative costs fi’om program costs, and recruitment data that leads to ques- 
tions about the program design and effectiveness. 

Recommendation 2: 

The Department should develop and implement a system of internal controls to provide 
reasonable assurance that adequate monitoring and oversight of contracted MEP activities 
will be provided and that the Department complies with applicable federal laws. 
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APPENDIX A 



Analysis of Migrant Education Program Evaluations 
and Performance Measurements 

As with any government program, the Department should periodically evaluate UVM’s performance to 
determine the effectiveness and efficiency of the program. Like many other states, Vermont has recently 
introduced a statewide system of standardized skills assessment tests for several grade levels. Previously, 
there was no reliable way to measure and compare student performance across supervisory districts. The 
most common indicators used were teacher’s assessments of whether students were working at or below 
grade level in basic skills and whether the students achieved low academic grades. However, there were 
no objective standards for these assessments. Standardized testing was implemented in 1998, but because 
the schools did not disaggregate the data, evaluating the performance of MEP students was not possible. 
As a result of Act 60, however, the State can now require schools to track MEP students and the 1999 
results should be disaggregated. 

Even when results for MEP students are available, “MEP’s effects on educational outcomes are difficult 
to measure because MEP funds are used in combination with funds from state, local, and other federal 
programs to achieve common educational goals. The relatively small size of MEP’s contribution adds to 
this difficulty. Nationally, it is estimated that MEP provides an average of $400 per student, which 
constitutes only a small fraction of the resources that a participant typically receives through state and 
local school programs.”* 

The MEP commissioned two (2) independent evaluations in the past five (5) years. 

A. 1995 Evaluation : In 1995, UVM’s College of Education and Social Services conducted an evaluation 
of the MEP. The methodology included interviews with a small sample of students, their parents, 
teachers and school administrators, as well as a review of additional school records. The researchers 
found that the MEP was “well integrated with the regular school program” and that “the Migrant 
teacher is part of a team to help migrant students achieve educational objectives which are arrived at in 
partnership between migrant teachers, regular teachers and others.”^ The program was found to be 
“systematic and well organized” and that “when the system functions smoothly the results are 
impressive.” The report noted that because “learning and behavioral objectives are of necessity varied 
and unstandardized . . . norm referenced standardized tests are inappropriate measures of pupil 
outcomes. To the extent that they are used by state officials to index program success they will result 
in incomplete and flawed conclusions of program performance.” Additionally, “the fact that the 
program is designed to be a short-term, supplemental intervention mitigates to some extent against 
longitudinal design for program evaluation with more standard measures.” The report recommended 
against evaluating MEP students using the standardized tests that are now required throughout the 



' “Child Labor in Agriculture: Changes Needed to Better Protect Health and Educational Opportunities,” U.S. General 
Accoimting Office, August 1998, p. 58. 

^ Meyers, H.W., “1993-94 Evaluation Report for the Migrant Education Program of the Office of Rural Education,” College 
of Education and Social Services, UVM, March 19, 1995 (revised Febmary 12, 1996), p. 14. 
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State. The report suggested that it would be more appropriate “to continue independently conducted 
case studies of randomly drawn cases in order to verify the levels of performance.” 

Use of non-standardized evaluative criteria may require more work for teachers. They must assess 
student needs, develop educational plans with clear objectives, design and administer evaluative 
activities, and keep good records. The goal in the State plan was that “80 % of the students will meet 
at least 75 % of the planned objectives set for them.” The researchers found that “the case studies and 
record reviews [were] sufficient to estimate in general that as many or more than two thirds of the 
students made important progress on educational plans.” However, there was a “lack of specific 
objectives, criteria, and measures in about a third of student records [so] it [was] not possible to 
evaluate the objective as written.” The 1995 study did not examine or evaluate the administrative 
structure, budget, or efficiency of the MEP. 

B. 1998 Evaluation : The MEP contracted with an outside consultant to evaluate the program in 1998. 

The goals and methodology included 1) documentation of services (tabulate Family Contact Logs and 
Family-Teacher Compacts); 2) monitoring of progress (review migrant teachers comments fi’om staff 
meetings); and 3) feedback on program objectives (review staff assessments and parent survey results). 
Unlike the 1995 evaluation, no students or parents were interviewed and the researchers relied almost 
exclusively on information supplied by program staff The report concluded that the MEP had “been 
very successful in reaching a large number of migrant families and identifying and meeting their needs 

» » » 'X 

for supportive educational and social services.” After reviewing the report we identified several areas 
where the methodology was questionable and the data may or may not support the conclusions. 

1 . Family Contact Logs and Family-Parent Compacts : While a review of Family Contact Logs 
documents efforts made by migrant teachers and staff, it cannot, tell us anything about the nature, 
substance, or effect of the communication. The logs, which were used to determine the average 
number of contacts per family, included visits and phone calls when the adult was not home but a 
message was left. The logs also include contacts with fiiends, baby sitters, and “other (non- 
instructional) school staff.” 

The logs also were used to determine the number and percentage of families with particular 
concerns (e.g., attendance, performance, readiness, family problems, etc.) and the number and 
percentage with positive outcomes. Overall, the outcomes reported demonstrated progress in 
several areas but we were not able to gauge the degree of improvement because no benchmarks or 
gradations were provided. For example, positive outcomes for performance, attendance, and 
readiness were characterized as “improved.” Although less than half the families with these 
concerns were found to have had positive outcomes. Finally, it is not possible to know if the 
percentage of those showing “improvement” was commendable because no information was 
provided about the performance of MEP students in prior years or in other states, or how non-MEP 
students fared. 

The report also listed 214 goals established in Family-Teacher Compacts but failed to indicate how 
many families (out of 541) agreed to the compacts or how many students had multiple goals. 

Using information fi’om the teachers’ files, 33 percent of the goals were achieved, there was some 



^ “Vermont Migrant Education Program Evaluation Report,” Robert V. Carlson, ABC Associates, November 6, 1998, p. 12. 
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progress on 43 percent of the goals, and for the remaining 24 percent there was little or no progress 
or no response. These results are encouraging and suggest that, once engaged, teachers and parents 
can achieve results. However, it’s not possible to know how many students benefited without more 
data. Moreover, most goals were subjective with no criteria provided so it’s difficult to know what 
“completion” meant (e.g., “address student’s personal behavior”). For those goals more 
quantitative in nature (e.g., improve reading and math skills, etc.), it should have been possible to 
measure the level of improvement through grades and other assessment methods. However, the 
report provided no supporting data. 

2. Program Objectives and Indicators : To evaluate whether the MEP met its program objectives, 
the researchers reviewed a report prepared by MEP staff, which was reproduced in the evaluation. 
However, there is no evidence that the researchers attempted to verify the accuracy or reliability of 
the data provided. Moreover, the evaluator failed to comment on the appropriateness of the 
indicators. This is significant because we found several that were not outcome measures. 

Our review of the staff report contradicts the findings of the evaluation. Of the nine (9) broad 
objectives, the consultant reported that the MEP had met or exceeded six (6), that one (1) was 
partially met, and that there was insufficient data to evaluate the other two (2). In contrast, we 
found that of the twenty-one individual goals, eight were not outcome measures, four were partially 
met, there was insufficient information for four, two had no data whatsoever and three were not 
met [see Appendix B for a detailed review of this section of the evaluation]. 

3. Parent Survey : The Department sent a parent survey to 536 MEP families and 105 families, or (20 
percent), responded. There is no evidence that the Department attempted to elicit additional 
responses with follow-up mailings or phone calls. No information was requested about the 
characteristics of the families surveyed so we have no information about whether the sample was 
representative of the entire universe of Vermont MEP families. Of those who responded, 80 
percent stated their experience with the MEP as excellent, very good or good. 

The survey also included seven (7) other questions (four “yes” or “no” and three open-ended) but 
the Department did not tabulate or report the results. According to Department staff, the number of 
responses were limited and tended to confirm the finding that overall support was strong.'* Finally, 
some of the responses to the open-ended questions were not entirely clear and were of limited 
analytical value. 

4. Staff Comments : The evaluation contractor decided to review and summarize comments from 
teachers recorded at four (4) staff meetings over the course of a ye^. The teachers provided 
insight into their work, the problems facing migrant families, and the challenge of coordinating 
services for the students. While this type of anecdotal information from teachers is of considerable 
value for program design, resource allocation, program integration, and organizational 
development, most of it is not useful for a performance evaluation of the program. It is noteworthy 
that the contractor, who did not attend the staff meetings, listened to tape recordings so he was not 
able to ask follow-up questions about their comments. Finally, since program administrators were 



April 27, 1999 telephone conversation between Mary Mulloy, Consultant, VT DOE and Doug Hoffer, Consultant, KPMG 
contractor. 
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at the meetings, it is possible that some teachers may have been reluctant to speak freely or criticize 
the program. 

C. Annual State Plans : Typically, each aimual State plan includes a section about evaluation. The FY 97 
and FY 99 plans contained evaluation sections but, as was noted above, there was no State plan in FY 
98. We reviewed the evaluation sections in the FY 97 and FY 99 State plans and found that almost no 
performance data was reported. For example, the FY 97 plan described a three-tiered approach that 
was “in the process of development.” The plan was to include the state standardized tests, an internal 
organizational evaluation, and data collection on staff contact with and service delivery to families and 
students. 

The FY 99 State plan provided no follow-up regarding the elements of the proposed three-tiered 
approach. Instead, it described a new methodology that bore some resemblance to the three-tiered 
approach of FY 1997 but, once again, contained no data. However, the 1998 evaluation (released after 
the FY 99 plan and reviewed above) included information about Family Contact Logs and Family- 
Teacher Compacts. 

The annual State plans contain information from student assessments but they cannot be considered 
evaluations of the program. For example, about a third of the information reported is demographic and 
descriptive (e.g., health problems, family background, etc.), while another third provides information 
about the student’s eligibility for other special education services (e.g., speech and language. Title 1, 
etc.). Several measures relate to the student’s performance in school (e.g., percentage reading at grade 
level, academic grades, etc.) and are presented along with data from prior years. The data collection 
process was changed in FY 97 so the data is not comparable. There is no assessment data from FY 98 
(the year no State plan was produced). If the methodology were sound and consistent, the assessment 
data would be an important part of a program evaluation. However, given the number of students that 
enter and leave the program annually, the analysis would need to differentiate between students who 
had received services for more than one year and those who were new to the program. 

There is at least one source of comparative performance data not utilized by the Department or the 
MEP. The U.S. Department of Education’s national MEP collects data from the states regarding 
iimuts, number of students served, services offered, and the number and type of staff [see Tables 1 - 
3^]. Although there are many differences between states, this information can be useful. For example, 
there are at least three key 
measures that are available 
but have never been 
presented by the MEP or the 
Department. They are 1) 
cost per student; 2) 
percentage of instructional 
vs. non-instructional staff 
and the ratio of students to 
staff; and 3) the types of 
services provided. While 



Table 1 



State 


No. of 
Students 


FY97 

Grant 


f Cpst per 
f Student 


W 


1,305 


$909,980 


$697 


Maine 


9,791 


$3,359,037 


$343 


Mass. 


4,174 


$2,728,433 


$654 


Conn. 


5,237 


$2,162,837 


$413 


NH 


119 


$95,738 


$805 



^ All data in Tables 1 - 3 from the US Department of Education, Migrant Education Program. 
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these measures cannot tell us how effective a state program may be in meeting student needs; they can 
allow comparisons about how resources are allocated. Although cost per student is interesting (Table 
1), it must be combined with output and outcome data to be useful. 



Table 2 





Regular Term 




Instructional Staff 


Non-instructional Staff 


State 


%of 

staff 


Ratio 


%of 

staff 


Ratio 


VT 


45 % 


114to 1 


55% 


91 to 1 


Maine 


94% 


78 to 1 


6% 


1130 to 1 


Mass. 


22% 


688 to 1 


78% 


192 to 1 


Conn. 


38% 


21 1 to 1 


62% 


129 to 1 


NH 


100% 


60 to 1 


0% 


NA 



For example, Vermont’s ratio of regular term instructors to students during the 1996-97 school year 
placed it in the middle of the pack for New England (Table 2). However, Vermont’s ratio of non- 
instructional staff to students was the lowest in New England, which meant that we had considerably 
more non-instructional staff per student than other states. 

The picture is not complete imtil we know more about the tasks assigned to instructional staff. As can 
be seen in Table 3, Vermont MEP students received virtually no direct instruction in any of the major 
skill and subject areas during the 1996-97 school year (e.g., reading, math, science, etc.). 
Massachusetts was similar in this regard while Maine and New Hampshire offered instruction to a 
much higher percentage of students. Only during the summer term did Vermont’s MEP offer direct 
instruction to a significant percentage of students. Even so, Vermont was well below the national 
average in this regard for all skills except language arts. The advocacy model favored by Vermont is 
reflected in the fact that the primary function of our instructors was social work rather than teaching. 
The only other supporting service provided in Vermont was health but MEP nurses have since been 
laid off. 
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Table 3 



Percentage of Migrant Students Receiving Services 96-97 




VT 


Maine 


Mass. 


Conn. 


NH 


National 

Avg. 




Regular 

Term 


Summer 

Term 


Regular 

Term 


Summer 

Term 


Regular 

Term 


Summer 

Term 


Regular 

Term 


Summer 

Term 


Regular 

Term 


Summer 

Term 


Regular 

Term 


Summer 

Term 


Instructional 














ESL 


0.0 


0.0 


5.0 


2.0 


0.0 


0.0 


4.8 


3.4 


1.7 


1.3 


15.3 


15.3 


Reading 


2.5 


8,6 


85.5 


89.8 


0.0 


0.0 


4.0 


22.22 


93.3 


98.7 


24.7 


42.3 


Lang. Arts 


0.5 


96.3 


43.4 


58.9 


3.6 


85.3 


4.4 


32.6 


14.2 


22.7 


19.5 


34.5 


Math 


1.3 


15.1 


27.9 


40.1 


0.0 


0.0 


2.5 


1.4 


13.3 


18.7 


22.3 


38.0 


Vocational 


0.6 


0.0 


26.8 


53.3 


0.0 


1.5 


12.8 


0.2 


4.2 


0.0 


6.0 


6.8 


Soc. 

Studies 


0.2 


0.0 


13.9 


15.8 


0.0 


0.0 


0.2 


0.0 


5.8 


6.7 


8.2 


10.1 


Science 


0.0 


1.1 


22.4 


56.8 


0.0 


0.0 


0.1 


1.0 


93.3 


98.7 


6.9 


14.2 


other 


1.6 


0.0 


35.7 


44.2 


4.3 


53.6 


12.7 


60.1 


0.8 


21.3 


26.7 


47.4 


Supporting 














Guidance 


3.0 


0.0 


13.1 


12.0 


0.0 


0.0 


2.9 


0.0 


6.7 


10.7 


29.7 


24.8 


Social 

Work 


100 


100 


67.0 


63.7 


100 


100 


99.3 


26.3 


56.7 


54.7 


44.3 


29.1 


Health 


36. 

2 


9.2 


26.7 


37.8 


0.0 


83.3 


3.3 


0.4 


0.0 


0.0 


11.6 


12.4 


Dental 


0.0 


0.0 


0.0 


0.0 


0.0 


49.2 


2.7 


0.0 


0.0 


0.0 


6.7 


8.8 


Nutrition 


0.0 


0.0 


3.2 


65.4 


0.0 


100 


3.3 


0.9 


0.0 


0.0 


8.2 


23.2 


Pupil Trans 


0.4 


0.0 


4.3 


55.9 


0.3 


100 


4.5 


8.5 


0.0 


4.0 


9.3 


29.3 


Other 


14. 

7 


95.9 


68.1 


73.4 


0.0 


0.0 


0.0 


0.0 


0.0 


26.7 


24.4 


20.3 



Note: The data presented above must be viewed with a certain amount of caution. First, the data is 
self-reported and has not been audited. Second, the federal categories and definitions may not capture 
subtle differences between states. Third, there may be significant differences between the target 
populations, although such differences would tend to be less dramatic within New England than 
between regions. Nevertheless, the data provides an overview of Vermont’s resource allocation 
strategy and addresses portions of two key components of performance measurement (inputs and 
outputs) not presented by either the MEP or the Department. 

D. 32 V.S.A. §307(cl ; In 1994, the Legislature amended 32 V.S.A. §307 to include subsection (c) that 
requires the inclusion of a strategic plan in all State government budgets. Among other things, the plan 
must include a “description of indicators used to measure output and outcome.” One important 
purpose of the performance information is to inform and support the Legislature’s budget decisions. 
Since the MEP receives no State fimds, one could argue that some of the justification for the 
performance data is lacking. However, regardless of the source of the money, the Department of 
Education has an obligation to ensure that programs supported with taxpayer fimds are efficient and 
effective. 



er|c 
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We reviewed the Department’s Strategic Overview (Form 4) in its FY 2000 budget submission and 
foimd that the only information provided about the MEP was the number of students in the program. 

Comment : Even though the statute requires it, the Department has never insisted that the MEP 
measure performance. This problem is not unique to Vermont, however. In a recent report to 
Congress, the General Accoimting Office foimd that “no program data are available to . . . measure 
MEP results. [The Department of] Education collects a considerable amount of information from the 
states on MEP participation, staffing, and services, and has for many years, but these data cannot be 
used to measure program accomplishments, or states’ progress in meeting national service 
objectives.”® 

There is no doubt that measuring the performance of a program like the MEP is a challenge. As was 
noted, there have been no objective criteria that could be applied consistently across school districts 
(although some argue that standardized tests are not the most appropriate evaluative method for special 
needs students). In addition to migrancy, there are numerous environmental factors that may 
contribute to poor student performance in school (e.g., poverty, health, etc.) and confound efforts to 
isolate causes and effects. The lack of reliable data from other states makes it impossible to compare 
the performance of Vermont’s MEP with other jurisdictions. Finally, the nature and quality of general 
(non-MEP) instruction and guidance varies across school districts and MEP is only a small part of the 
total expended per student. Nevertheless, the Govermnent Accounting Standards Board (GASB) has 
developed a reasonable framework for performance measurement that can help guide the Department.^ 



* “Child Labor in Agriculture: Changes Needed to Better Protect Health and Educational Opportunities,” U.S. General 
Accounting Office, August 1998, pp. 58-59. 

’ “Service Efforts and Accomplishments Reporting,” Concepts Statement No. 2, Governmental Accounting Standards Board, 
No. 109-A, April 1994. 
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APPENDIX B 



Analysis of Migrant Education Program Objectives and Indicators ' 

Since we commented on the lack of useful or consistent performance measures, we decided to review in detail the 
goals and indicators contained in the MEP’s 1997-98 “Program Objectives Results.” This document represents the 
sole source of quantitative information about program performance in the last five (5) years. As was noted above, it 
was one of three major elements of the 1998 independent evaluation (although the data contained in the report was 
not independently verified by the consultant). All quotations without citations are from the report.' 

Goal 1: “To ensure that all migrant children are treated equitably in all schools . . . and that they have full access to 
all needed education programs and services.” 

> Indicator of Success : ’’Twelve (12) teachers will offer a statewide service of student advocacy and coordination 
for 100 % of migrant students. ” 

Goal 1 is a restatement of the program’s mission and is not specific or measurable (one cannot reliably measure 
equitable treatment). This indicator does not relate directly to the stated goal and is actually just a declaration of. 
the budgeted number of positions and is, therefore, an input measure rather than an outcome. 

> Indicator of Success : ’’Twenty (20) principals with a significant number of migrant students will be contacted by 
the Director and Assistant Director to discuss equitable treatment and full access. ” 

Although this effort is a necessary precondition for attainment of the goal, it is not an outcome. 

Goal 2: “To provide continuity in education experience for migrant children moving from school to school within 
the state and to or from schools in other states.” 

>• Indicator of Success : ”100 % of instate moves will be anticipated with or followed up with a transfer of school 
and migrant records. " 

The report stated that “MEP teachers transfer records among themselves as appropriate.” No data was provided 
to substantiate this statement. 

>• Indicator of Success : ”100 % of children moving out of state will have their Certificate of Eligibility (COE) 
faxed to the receiving school. ” 

The report stated that “COE’s are faxed to state MEP departments in other states.” No data was provided to 
substantiate the success of this indicator. 

Indicator of Success: ”100 % of students entering the state will have their records located and transferred as 
necessary.” 

The report indicated that staff conduct an identification and recruitment visit if notified by the MEP in the state 
of last residence but that the transfer of records is up to the schools. In addition, the report noted that the 
database was “not configured to provide the documentation relative to the indicator.” 



' Carlson, Robert, Vermont MEP Evaluation Report, Attachment C, November 6, 1998. 
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Most Vermont MEP families move within the State but some come from neighboring states. The MEP 
participates in the Eastern Stream Advance Notification System (ESANS) that tracks the number of notifications 
sent to and received from other states. With this information, MEP program administrators could have 
established a simple protocol with the schools to ensure the prompt transfer of records. Unfortunately, “in 
Vermont, the definition of school records is left up to the local school . . . and some schools do not include any 
Special Education information.”^ Thus, while it may be appropriate for the schools to handle the actual transfer, 
the issue is important enough that MEP staff should be expected to monitor individual cases. The report offered 
no information about the effectiveness or timeliness of the school’s efforts. 

Goal 3: “To ensure that all migrant children have an opportunity to enter school ready to learn.” 

> Indicator of Success : "100 % of pre-school migrant children will be checked for pre-school screening. " 

Only 49 of 151 pre-school children (32 percent) were screened. The narrative noted that the number was not 
surprising because some pre-school children may have been infants but failed to say how many actually were 
infants. As a result, we have no idea what percentages of non-infant pre-school children were screened. 
Furthermore, the MEP’s FY 99 grant application to the state (which used data from the previous school year) 
reported that assessments were completed for only 79 % of pre-school children in 1 996-97. 

> Indicator of Success : "100 % of parents with children entering school for the first time will be offered services 
to facilitate communication with the school. " 

According to the report, eighty-four (84) kindergarten students received home visits and the narrative counted 
this as a goal that had been met. But the report did not describe the services offered during the home visit nor 
did it indicate the total number of kindergarten students. We learned subsequently from MEP staff that there 
were eighty-eight (88) MEP kindergarten students.^ Therefore, since only 95 percent of kindergarten students 
received home visits, the goal was not met. 

> Indicator ofrSuccess .' "A pre-school child will be considered for migrant direct instruction if unable to 
participate in existing programs. ” 

The report indicated that sixty-nine (69) children were participating in other existing pre-school programs and 
ninety (90) were not. However, the narrative contained no information about whether the 90 children not 
participating in other existing programs needed or were eligible for migrant direct instruction. We learned 
subsequently that not all the other ninety (90) received direct instruction.'* * We understand that federal guidelines 
require the MEP to prioritize services to those that are failing or at most risk of failing, and whose education has 
been interrupted by moves during the regular school year. Uie indicator recognized this by calling for 
“consideration” for direct instruction. But consideration for eligibility is not an outcome measure and the report 
contained no data to support the contention that the goal had been met. 

Goal 4: “To reduce the incidence of dropouts and improve the graduation rate among migrant students.” 

y Indicator of Success : "60 % of known out-of-school migrant youth will be offered guidance to diploma / GED 
programs or job training. ” 



^ April 15, 1999 memorandum from Shirley Wolfe, Acting MEP Director, to Doug Hoffer, Consultant, KPMG contractor. 
^ April 15, 1999 letter from Shirley Wolfe, Acting Dir. of MEP, to Doug Hoffer, Consultant, KPMG contractor. 

* April 15, 1999 letter from Shirley Wolfe to Doug Hoffer. 
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No explanation was provided as to why the goal was set at 60 percent rather than 100 percent. In light of the 
mobility of MEP families, it is not surprising that some out-of-school youth would be difficult to locate. 
Nevertheless, the goal should have been to reach all of them. The report indicated that 12 of 17 out-of-school 
youth (70 percent) were contacted and that three (3) had completed a GED, four (4) were enrolled in GED 
courses, staff were still in contact with five (5), and five (5) others were not reachable. While this is 
encouraging, it refers only to students who have already left school. The first part of the stated goal was to 
reduce the incidence of dropouts. No information was provided in the report about the historical graduation rate 
for migrant students in Vermont or elsewhere, or a comparable rate for non-MEP students. 

Goal 5: “To provide migrant children a wide range of education opportunities.” 

> Tndictor of Success : "100 % of migrant students will have available, relative to their needs, home-school 
advocacy, agency coordination, and instructional service; summer program; health coordination and referral; 
family literacy; and College Assistance Migrant Program. ” 

The report stated that “definitive reports reflective of data needed for [this indicator] are not available until 
November.” This highlights the need for better data collection and better coordination between the MEP and 
other State-fimded programs. 

Goal 6: “To provide specially targeted educational services not available from other services.” 

> Indicator of Success: Same as U5. 

The report contained no explanation. Presumably, there was no data available. 

Goal 7: “To empower migrant parents to become active partners in the education of their children.” 

> Indicator of Success : "Convene four (4) regional parent-staff meetings. ” 

The report noted that eight (8) meetings were held in FY 98. While organizing such meetings may be one of 
many appropriate strategies for achieving the goal, empowerment cannot be measured by attendance at a 
meeting. Moreover, simply reporting the number of meetings tells us nothing about how many parents attended, 
what was discussed and what, if any, follow-up occurred - that is, were they successful meetings. Without more 
information, this is simply an output measure because it refers to actions (work) by MEP staff rather than 
measurable outcomes related to parents’ participation in their children’s education. 

> Indicator of Success : "Convene a May workshop of parents to review and amend an annual set of program 
recommendations. ” 

The report stated that a meeting was held in Newport on May 14, 1998. Again, this is not an outcome measure 
and, in any case, no information was provided about attendance or results of the meeting (e.g., did the MEP 
accept or act upon any parent comments or suggestions?). 

> Indicator of Success : "Representative group of parents [will] meet with Vermont Congressional staff to discuss 
legislative concerns. ” 

The report stated that five (5) parents met with congressional staff. It may be useful (and empowering) for 
migrant parents to meet with congressional staff but in itself, that tells us nothing about whether they became 
“active partners in the education of their children.” Furthermore, taking five parents (out of over 500 families) 
to meet congressional staff is not a programmatic effort. 
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> Indicator of Success : "Plan and participate in summer interstate parent workshop. ” 

According to the report, the workshop was canceled by the host state of Maine. 

Goal 8; “Training and skill development of migrant educators.” 

> Indicator of Success : "Implement a schedule of monthly staff meetings which will include 20 hours of 
professional development related to program needs. ” 

The report claimed this goal was met. Although professional development is an essential element of any service 
delivery program, it is not an outcome per se. It is more appropriately characterized as an input since it is 
intended to improve the effectiveness of the staff. Moreover, the fact that the goal included the 
“implementation” of monthly staff meetings and professional development implied that they had not previously 
occurred. If so, it is noteworthy since the program is thirty (30) years old. 

^ Indicator of Success : "New teachers will have regional team support and individual mentoring from program 
specialists. ” 

Again, this is not an outcome measure. 

> Indicator of Success : "Parent Compact will document agreed upon goals authorized by [the] parent and Rural 
Education teacher. Seventy five % (75 %) of parents will participate. ” 

The narrative stated that “an estimate of twenty % (20 %) was achieved” (emphasis added). It is not clear why 
this goal was not met because the report referred only to a problem of “staff and family readiness.” 

Goal 9: “To support education for migrant children through outreach, advocacy, linkage with service providers, and 
dissemination of information.” 

^ Indicator of Success : "Two migrant newsletters will be published and disseminated statewide and nationally. ” 

The report noted that one newsletter was published but budget constraints prohibited further publication. 

^ Indicator of Success : "Critical Friends ” of migrant education will meet twice to provide program guidance, 
support, and multi-level interaction and coordination. ” 

The report indicated that the group was convened only once. While such a group could be of service to the 
program, the mere fact of the meeting(s) would not in itself be an outcome. However, if the meetings led to 
improved coordination between various educational programs (as is required by statute), it would constitute a 
measurable outcome. 

> Indicator of Success : "The program will create an informal partnership with four service providers to mutually 
share and develop reciprocal working relationships. ” 

The report described only one partnership. See our comment in the preceding paragraph. 
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STATE OF VERMONT 
DEPARTMENT OF EDUCATION 
120 SUte Street 
Montpelier, VT 05620-2501 

March 24, 2000 



Michael J. Clasen 
Deputy State Auditor 
133 State Street 
Montpelier, VT 05633 

Dear Mr. Clasen; 



Thank you for your review of the Vermont Migrant Education Program. The Department is committed to 
meeting the educational needs of Vermont’s migrant children and fulfilling its obligations under state 
and federal law to manage federal migrant education funds properly. 

While I am in basic agreement with the findings and recommendations of the report, there are two 
omissions. With respect to Finding 1, that the Vermont migrant education plan did not meet federal 
requirements, it would seem pertinent to me to point out that the Federal Office of Migrant Education 
accepted our plan. 

Secondly, while the report makes note of the performance stipulations attached to the last UVM grant and 
the failure to follow-up on those stipulations, it does not offer any explanation for the lack of follow-up. 

In between the setting of the stipulations and the due dates for them, the Department and University 
mutually agreed to discontinue to operate the program through the University. In short, once that 
decision was made, compliance with the stipulations was moot. 

Thank you for the opportxmity to clarify these two issues. 

Sincerely, 

David Wolk, Commissioner 
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